Benchmarking

Benchmarking is Solid's built-in accuracy measurement system for semantic models.

The Five-Step Benchmark Process

1. Auto-generate questions — Solid creates benchmark questions from real SQL log patterns.
2. Attach ground truth — certified SQLs from query history are linked as expected answers for each question.
3. Modelers review questions — add, remove, or edit questions and their ground-truth SQLs before running.
4. Run benchmark — Solid generates SQL for each benchmark question and executes both the generated SQL and the certified ground-truth SQL.
5. Accuracy score calculated — results matching is the primary pass signal. SQL structure and metadata used are also evaluated; both aligning adds confidence to the pass.

Pass Criteria

• Primary signal: the results returned by Solid's generated SQL match the results returned by the certified ground-truth SQL.
• Confidence booster (secondary): SQL structure is similar, and the metadata used (tables, columns, joins) aligns with the semantic model. Both aligning raises confidence in the pass.

Accuracy Score

The percentage of benchmark questions that pass. Target accuracy for pilot use cases is 85–90%. After applying fix recommendations iteratively, accuracy can reach 97%+ — see SQL Generation and Grounding for the correction loop that gets you there.

Benchmark Health

Benchmark health scores the quality of the benchmark itself, before you trust the accuracy number it produces. A flawed benchmark can mislead you about real model quality.

For each benchmark health dimension failure, Solid surfaces a targeted recommendation with an "Apply Fix — 1 Click" action — fix the benchmark's own quality before using its accuracy score to make model decisions.

Model Coverage

A high accuracy score only tells part of the story — it also matters how much of the model your benchmark actually validates. A model can show 90% accuracy while large sections of it are completely untested.

The Benchmark Questions page shows:

Coverage is calculated from the expected SQL of active benchmark questions — the certified ground truth — not from what the model generates. This means coverage reflects the intended scope of the benchmark, not an artifact of model quality.

Each benchmark question also shows which model assets (tables, columns, metrics) its expected SQL references, making it easy to see exactly how each question contributes to overall coverage. You can also search benchmark questions by table or column name, not just by question text.

Closing coverage gaps

When you identify uncovered columns or tables, Solid can auto-generate targeted benchmark questions for those gaps. Review the suggestions and add relevant ones directly to your benchmark suite — this creates a direct loop:

Find coverage gaps → auto-generate benchmark questions → choose relevant questions → validate accuracy with more confidence.

Benchmark Optimize

After a benchmark run, Solid analyzes failures and surfaces a ranked list of actionable recommendations — ranked by how many questions each fix is expected to unblock. Each recommendation shows the fix type (Model Fix, Instruction Fix, or Benchmark Fix) and the affected questions. Click "Fix with AI" to preview and apply a change in one click. Recommendations are conflict-checked before surfacing, so no two suggestions contradict each other.

Why This Matters

Without benchmarking:

• There's no way to measure if a semantic model is accurate
• Regressions go undetected — a model that worked last month may fail silently
• Model quality is subjective and tribal-knowledge-dependent

Solid re-tests accuracy on every model change, rather than leaving models untested after the initial build.